Information retrieval (IR) is the process of obtaining relevant resources from a collection of information sources. Automated IR systems, such as web search engines, are often used to process user queries for resources (e.g., web pages, documents, etc.). Traditionally, IR systems have been evaluated in terms of the relevance of the resource result sets retrieved for individual queries. Recent research, however, has improved this evaluation by exploring the preference judgments for resources retrieved for multiple resource result sets for the same or similar user queries, referred to as Side-by-Side (SBS) evaluation. Currently, SBS evaluation requires a substantial resource investment and highly-trained and/or consistently-monitored judges to produce accurate results.
It is with respect to these and other general considerations that the aspects disclosed herein have been made. Also, although relatively specific problems may be discussed, it should be understood that the examples should not be limited to solving the specific problems identified in the background or elsewhere in this disclosure.